System and method for 3d tracking for ad-hoc cross-device interaction

ABSTRACT

Disclosed is a three-dimensional ad-hoc tracking system and method between two or more devices at a location. The disclosed systems and methods can be implemented on commodity mobile devices with no added components and require no cross-device calibration, in order to track surrounding devices. Such tracking can be achieved by fusing three types of signals: 1) the strength of Bluetooth low energy signals reveals the presence and rough distance between devices; 2) a series of inaudible acoustic signals and the difference in their arrival times produces a set of accurate distances between devices, from which 3D offsets between the devices can be derived; and 3) the integration of dead reckoning from the orientation and acceleration sensors of all devices at a location to refine the estimate and to support quick interactions between devices. The disclosed systems and methods can be implemented by cross-device applications on mobile devices.

This application includes material that is subject to copyrightprotection. The copyright owner has no objection to the facsimilereproduction by anyone of the patent disclosure, as it appears in thePatent and Trademark Office files or records, but otherwise reserves allcopyright rights whatsoever.

FIELD

The present disclosure relates generally to spatial awareness detection,and more particularly towards systems and methods for determiningthree-dimensional (3D) spatial presence of users at a location.

BACKGROUND

Bluetooth low energy (BLE) and radio technology can be used for robustlocation tracking. For example, such technology can be used incommercial situations to augment shopping trips in malls and airports byoffering relevant suggestions or coupons, or to support experiences instadiums during sports events. In order to track devices based on BLE orradio signals, conventional tracking systems interpolate the signalstrengths observed for each device, essentially using signal strength asa distance cue.

SUMMARY

The present disclosure describes systems and methods forthree-dimensional (3D) ad-hoc tracking between two or more devices at alocation. According to some embodiments, the disclosed systems andmethods can be implemented on any known or to be known commodity mobiledevice without the need for any additional components or cross-deviceapplication. Indeed, the disclosed systems and methods require nocross-device calibration. The present disclosure involves systems andmethods for tracking surrounding devices at a location by fusing threetypes of signals in order to track the surrounding devices. Suchtracking can be achieved by fusing three types of signals communicatedto and from such devices at the location. According to some embodiments,the fused signals include: 1) the strength of BLE signals of eachdevice; 2) inaudible acoustic signals of each device; and 3) deadreckoning of each device.

According to some embodiments, the BLE signals reveal the presence anddistance between each device at the location. In some embodiments, theinaudible acoustic signals comprise a series of radio signals, whereby adifference between each signal's arrival time is utilized to produce aset of accurate distances between each device at the location. The setof distances can be used to derive 3D offsets of/between each device atthe location. In some embodiments, the dead reckoning of each deviceinvolves integrating signals associated with orientation andacceleration sensors of each device at the location. The dead reckoningcan be used to refine the 3D tracking and spatial information of thedevices and location. The disclosed systems and methods involve eachdevice at a location sharing the above mentioned signal information,which can be propagated throughout the set of participating devices atthe location. The present disclosure enables the dissemination of thetracking information to peer devices beyond the reach of individual setsof devices.

In accordance with one or more embodiments, a method is disclosed whichincludes identifying, via a computing device, devices concurrentlypresent at a location in a structure; determining, via the computingdevice, a Bluetooth Low Energy (BLE) signal strength associated witheach identified device, the BLE signal strength determined from receivedBLE signals from each identified device; determining, via the computingdevice, an audio-distance measurement associated with each identifieddevice, the audio-distance measurement based on acoustic signals andtime stamps communicated to and from each identified device; receiving,at the computing device, an inertia measurement from each device; anddetermining, via the computing device, a three-dimensional (3D) spatialmapping of the location based on, for each identified device, the BLEsignal strength, the audio-distance measurement and the inertiameasurement, the 3D spatial mapping comprising identifiers andpositional information of each identified device within the location.

In accordance with one or more embodiments, a non-transitorycomputer-readable storage medium is provided, the computer-readablestorage medium tangibly storing thereon, or having tangibly encodedthereon, computer readable instructions that when executed cause atleast one processor to perform a method for determining users' 3Dspatial presence at a location.

In accordance with one or more embodiments, a system is provided thatcomprises one or more computing devices configured to providefunctionality in accordance with such embodiments. In accordance withone or more embodiments, functionality is embodied in steps of a methodperformed by at least one computing device. In accordance with one ormore embodiments, program code to implement functionality in accordancewith one or more such embodiments is embodied in, by and/or on anon-transitory computer-readable medium.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features, and advantages of thedisclosure will be apparent from the following description ofembodiments as illustrated in the accompanying drawings, in whichreference characters refer to the same parts throughout the variousviews. The drawings are not necessarily to scale, emphasis instead beingplaced upon illustrating principles of the disclosure:

FIG. 1 is a schematic diagram illustrating an example of communicationsystem according to some embodiments of the present disclosure;

FIG. 2 depicts is a schematic diagram illustrating a client device inaccordance with some embodiments of the present disclosure;

FIG. 3 is a schematic block diagram illustrating components of a systemin accordance with embodiments of the present disclosure;

FIG. 4 is a flowchart illustrating steps performed in accordance withsome embodiments of the present disclosure;

FIG. 5 is a flowchart illustrating steps performed in accordance withsome embodiments of the present disclosure;

FIG. 6 is a flowchart illustrating steps performed in accordance withsome embodiments of the present disclosure;

FIG. 7 is a flowchart illustrating steps performed in accordance withsome embodiments of the present disclosure;

FIG. 8 is a flowchart illustrating steps performed in accordance withsome embodiments of the present disclosure;

FIG. 9 is a flowchart illustrating steps performed in accordance withsome embodiments of the present disclosure;

FIG. 10 is a block diagram illustrating architecture of a hardwaredevice in accordance with one or more embodiments of the presentdisclosure;

FIG. 11 illustrates an example embodiment in accordance with one or moreembodiments of the present disclosure;

FIG. 12 illustrates a block diagram of a coordinate system in accordancewith some embodiments of the present disclosure; and

FIG. 13 illustrates a block diagram of a coordinate system in accordancewith some embodiments of the present disclosure.

DESCRIPTION OF EMBODIMENTS

The present disclosure will now be described more fully hereinafter withreference to the accompanying drawings, which form a part hereof, andwhich show, by way of illustration, specific example embodiments.Subject matter may, however, be embodied in a variety of different formsand, therefore, covered or claimed subject matter is intended to beconstrued as not being limited to any example embodiments set forthherein; example embodiments are provided merely to be illustrative.Likewise, a reasonably broad scope for claimed or covered subject matteris intended. Among other things, for example, subject matter may beembodied as methods, devices, components, or systems. Accordingly,embodiments may, for example, take the form of hardware, software,firmware or any combination thereof (other than software per se). Thefollowing detailed description is, therefore, not intended to be takenin a limiting sense.

Throughout the specification and claims, terms may have nuanced meaningssuggested or implied in context beyond an explicitly stated meaning.Likewise, the phrase “in one embodiment” as used herein does notnecessarily refer to the same embodiment and the phrase “in anotherembodiment” as used herein does not necessarily refer to a differentembodiment. It is intended, for example, that claimed subject matterinclude combinations of example embodiments in whole or in part.

In general, terminology may be understood at least in part from usage incontext. For example, terms, such as “and”, “or”, or “and/or,” as usedherein may include a variety of meanings that may depend at least inpart upon the context in which such terms are used. Typically, “or” ifused to associate a list, such as A, B or C, is intended to mean A, B,and C, here used in the inclusive sense, as well as A, B or C, here usedin the exclusive sense. In addition, the term “one or more” as usedherein, depending at least in part upon context, may be used to describeany feature, structure, or characteristic in a singular sense or may beused to describe combinations of features, structures or characteristicsin a plural sense. Similarly, terms, such as “a,” “an,” or “the,” again,may be understood to convey a singular usage or to convey a pluralusage, depending at least in part upon context. In addition, the term“based on” may be understood as not necessarily intended to convey anexclusive set of factors and may, instead, allow for existence ofadditional factors not necessarily expressly described, again, dependingat least in part on context.

The present disclosure is described below with reference to blockdiagrams and operational illustrations of methods and devices. It isunderstood that each block of the block diagrams or operationalillustrations, and combinations of blocks in the block diagrams oroperational illustrations, can be implemented by means of analog ordigital hardware and computer program instructions. These computerprogram instructions can be provided to a processor of a general purposecomputer, special purpose computer, ASIC, or other programmable dataprocessing apparatus, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, implement the functions/acts specified in the block diagramsor operational block or blocks. In some alternate implementations, thefunctions/acts noted in the blocks can occur out of the order noted inthe operational illustrations. For example, two blocks shown insuccession can in fact be executed substantially concurrently or theblocks can sometimes be executed in the reverse order, depending uponthe functionality/acts involved.

These computer program instructions can be provided to a processor of ageneral purpose computer, special purpose computer, ASIC, or otherprogrammable data processing apparatus, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, implement the functions/acts specified in theblock diagrams or operational block or blocks.

For the purposes of this disclosure a computer readable medium (orcomputer-readable storage medium/media) stores computer data, which datacan include computer program code (or computer-executable instructions)that is executable by a computer, in machine readable form. By way ofexample, and not limitation, a computer readable medium may comprisecomputer readable storage media, for tangible or fixed storage of data,or communication media for transient interpretation of code-containingsignals. Computer readable storage media, as used herein, refers tophysical or tangible storage (as opposed to signals) and includeswithout limitation volatile and non-volatile, removable andnon-removable media implemented in any method or technology for thetangible storage of information such as computer-readable instructions,data structures, program modules or other data. Computer readablestorage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM,flash memory or other solid state memory technology, CD-ROM, DVD, orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other physical ormaterial medium which can be used to tangibly store the desiredinformation or data or instructions and which can be accessed by acomputer or processor.

For the purposes of this disclosure the term “server” should beunderstood to refer to a service point which provides processing,database, and communication facilities. By way of example, and notlimitation, the term “server” can refer to a single, physical processorwith associated communications and data storage and database facilities,or it can refer to a networked or clustered complex of processors andassociated network and storage devices, as well as operating softwareand one or more database systems and application software that supportthe services provided by the server. Servers may vary widely inconfiguration or capabilities, but generally a server may include one ormore central processing units and memory. A server may also include oneor more mass storage devices, one or more power supplies, one or morewired or wireless network interfaces, one or more input/outputinterfaces, or one or more operating systems, such as Windows Server,Mac OS X, Unix, Linux, FreeBSD, or the like.

For the purposes of this disclosure a “network” should be understood torefer to a network that may couple devices so that communications may beexchanged, such as between a server and a client device or other typesof devices, including between wireless devices coupled via a wirelessnetwork, for example. A network may also include mass storage, such asnetwork attached storage (NAS), a storage area network (SAN), or otherforms of computer or machine readable media, for example. A network mayinclude the Internet, one or more local area networks (LANs), one ormore wide area networks (WANs), wire-line type connections, wirelesstype connections, cellular or any combination thereof. Likewise,sub-networks, which may employ differing architectures or may becompliant or compatible with differing protocols, may interoperatewithin a larger network. Various types of devices may, for example, bemade available to provide an interoperable capability for differingarchitectures or protocols. As one illustrative example, a router mayprovide a link between otherwise separate and independent LANs.

A communication link or channel may include, for example, analogtelephone lines, such as a twisted wire pair, a coaxial cable, full orfractional digital lines including T1, T2, T3, or T4 type lines,Integrated Services Digital Networks (ISDNs), Digital Subscriber Lines(DSLs), wireless links including satellite links, or other communicationlinks or channels, such as may be known to those skilled in the art.Furthermore, a computing device or other related electronic devices maybe remotely coupled to a network, such as via a telephone line or link,for example.

For purposes of this disclosure, a “wireless network” should beunderstood to couple client devices with a network. A wireless networkmay employ stand-alone ad-hoc networks, mesh networks, Wireless LAN(WLAN) networks, cellular networks, or the like. A wireless network mayfurther include a system of terminals, gateways, routers, or the likecoupled by wireless radio links, or the like, which may move freely,randomly or organize themselves arbitrarily, such that network topologymay change, at times even rapidly. A wireless network may further employa plurality of network access technologies, including Long TermEvolution (LTE), WLAN, Wireless Router (WR) mesh, or 2nd, 3rd, or 4thgeneration (2G, 3G, or 4G) cellular technology, or the like. Networkaccess technologies may enable wide area coverage for devices, such asclient devices with varying degrees of mobility, for example.

For example, a network may enable RF or wireless type communication viaone or more network access technologies, such as Global System forMobile communication (GSM), Universal Mobile Telecommunications System(UMTS), General Packet Radio Services (GPRS), Enhanced Data GSMEnvironment (EDGE), 3GPP Long Term Evolution (LTE), LTE Advanced,Wideband Code Division Multiple Access (WCDMA), Bluetooth, Bluetooth LowEnergy Technology (BLE) as a function of the Bluetooth CoreSpecification Version 4.0 of Bluetooth, 802.11b/g/n, or the like. Awireless network may include virtually any type of wirelesscommunication mechanism by which signals may be communicated betweendevices, such as a client device or a computing device, between orwithin a network, or the like.

A computing device may be capable of sending or receiving signals, suchas via a wired or wireless network, or may be capable of processing orstoring signals, such as in memory as physical memory states, and may,therefore, operate as a server. Thus, devices capable of operating as aserver may include, as examples, dedicated rack-mounted servers, desktopcomputers, laptop computers, set top boxes, integrated devices combiningvarious features, such as two or more features of the foregoing devices,or the like. Servers may vary widely in configuration or capabilities,but generally a server may include one or more central processing unitsand memory. A server may also include one or more mass storage devices,one or more power supplies, one or more wired or wireless networkinterfaces, one or more input/output interfaces, or one or moreoperating systems, such as Windows Server, Mac OS X, Unix, Linux,FreeBSD, or the like.

For purposes of this disclosure, a client (or consumer or user) devicemay include a computing device capable of sending or receiving signals,such as via a wired or a wireless network. A client device may, forexample, include a desktop computer or a portable device, such as acellular telephone, a smart phone, a display pager, a radio frequency(RF) device, an infrared (IR) device an Near Field Communication (NFC)device, a Personal Digital Assistant (PDA), a handheld computer, atablet computer, a laptop computer, a set top box, a wearable computer,an integrated device combining various features, such as features of theforgoing devices, or the like.

A client device may vary in terms of capabilities or features. Claimedsubject matter is intended to cover a wide range of potentialvariations. For example, a cell phone may include a numeric keypad or adisplay of limited functionality, such as a monochrome liquid crystaldisplay (LCD) for displaying text. In contrast, however, as anotherexample, a web-enabled client device may include one or more physical orvirtual keyboards, mass storage, one or more accelerometers, one or moregyroscopes, a compass, global positioning system (GPS) or otherlocation-identifying type capability, or a display with a high degree offunctionality, such as a touch-sensitive color 2D or 3D display, forexample.

A client device may include or may execute a variety of operatingsystems, including a personal computer operating system, such as aWindows, iOS or Linux, or a mobile operating system, such as iOS,Android, or Windows Mobile, or the like. A client device may include ormay execute a variety of possible applications, such as a clientsoftware application enabling communication with other devices, such ascommunicating one or more messages, such as via email, short messageservice (SMS), or multimedia message service (MMS), including via anetwork, such as a social network, including, for example, Facebook®,LinkedIn®, Twitter®, Flickr®, or Google+®, Instagram™, to provide only afew possible examples. A client device may also include or execute anapplication to communicate content, such as, for example, textualcontent, multimedia content, or the like. A client device may alsoinclude or execute an application to perform a variety of possibletasks, such as browsing, searching, playing various forms of content,including locally stored or streamed video, or games (such as fantasysports leagues). The foregoing is provided to illustrate that claimedsubject matter is intended to include a wide range of possible featuresor capabilities.

The principles described herein may be embodied in many different forms.By way of background, communication interactions with and betweendevices at a location typically involve devices establishing a notion ofsurrounding (or peer) devices' presence using Bluetooth® or WiFi®signals, which simply displays a list of detected surrounding devices(e.g., a list of available WiFi networks to connect to). Someconventional applications may use the strength of radio signals toestimate the distance to a surrounding device. However, the conventionalsystems and techniques are limited to one-dimensional spatialprocessing. That is, detection of spatial processing in such systemsonly produces a mere listing of detected devices. Such shortcomings failto account for the fact that cross-device interactions, and a user'sperception of such interactions, are spatial, in that they take place ina three-dimensional (3D) space.

The disclosed systems and methods remedy such shortcomings bydetermining a spatial awareness of surrounding devices in a 3D space,which includes the position of each device at a location, within astructure such as a building, shopping center, mall, stadium or otherdefinable enclosed or open space, and can include a distance anddirection of each device with reference to a specific device. That is,the disclosed systems and methods can sense users' presence and theirposition within a location (e.g., room, office, store) by determiningthe location of their mobile devices and laying out a 3D spatial mappingof the surrounding devices. The present disclosure enables ad-hoccross-device mapping and interactions, and requires no externalinfrastructure as it is readily executed on any known or to be knowncommodity wireless device on which applications can run and that areequipped with communication functions outlined herein. For example, asdiscussed in more detail below, the disclosed systems and methods canenable file sharing between devices at a location simply by a userswiping items in the direction of a detected recipient.

As discussed in more detail below, the disclosed systems and methods candetermine other devices' presence at a location from an aggregation ofsignal information. The signal information can be based on thosedevice's BLE signals. The signal information can also be based on adetermined distance and direction between devices from the difference inarrival times of multiple exchanged inaudible acoustic (or stereo)signals. Also, the signal information can be based on orientation and/ormotion information detected using inertial sensors that each device isstandardly equipped with or by detecting Doppler shift in detectedsignals. Thus, the disclosed systems and methods readily enable ad-hoc,spatial cross-device interaction between devices at a location withoutthe need for calibration by solely using components found in currentmobile devices.

The present disclosure will now be described more fully with referenceto the accompanying drawings, in which various embodiments of thedisclosure are shown. The disclosure may, however, be embodied in manydifferent forms and should not be construed as being limited to theembodiments set forth herein.

FIG. 1 is a diagram illustrating a spatial mapping system according tosome embodiments of the present disclosure. Not all the components maybe required to practice the disclosure, and variations in thearrangement and type of the components may be made without departingfrom the spirit or scope of the disclosure. The spatial mapping system,according to some embodiments of the present disclosure, may includemobile devices 200, 200 a, 200 b, and . . . 200 n located at and arounda location 100 (where device 200 n signifies that there can be anynumber of devices intermittently positioned or moving about thelocation).

According to some embodiments, as discussed below, the systems andmethods discussed herein can be supported by microphone-only devices,such as, for example, smart-watches linked to a mobile device, where thetwo devices (watch and phone, or other combinations of wearable devicessuch as glasses or one or more items of intelligent clothing) form onelogical unit. As discussed in more detail below, device 200 can havefunctionality related to BLE technology. The mobile device 200 may be aterminal for providing a user with a service via BLE communication withanother mobile device 200 a, 200 b, and . . . 200 n. For example, themobile device 200 may register and manage information about the otherdevices, for example, Identification (ID) information, in a memory.

By way of background, Bluetooth™ technology enables short-range wirelesscommunication, rather than having to use cables to connect devices toeach other. That is, for example, when Bluetooth wireless technology isimplemented in a cellular phone or a laptop computer, the cellular phoneor the laptop computer may be connected to a wireless communicationnetwork without having to make a cable connection. All types of digitaldevices, including smartphones, tablets, phablets, printers, PersonalDigital Assistants (PDAs), desktop computers, mobile devices andterminals, wearable devices or clothing, fax machines, keyboards, andjoysticks and the like, can be a part of a Bluetooth system. Bluetoothwireless technology can also be used to form an interface between anexisting data network and peripheral devices and form a special groupbetween devices which are located far from a fixed networkinfrastructure. Bluetooth technology provides a robust wirelessconnection based on quick recognition and by using a frequency hoppingmethod. A Bluetooth module avoids interference with other signals byhopping to a new frequency after the transmission or reception of apacket. Compared to other systems which operate within the samefrequency range, the Bluetooth technology uses an especially short andfast packet exchange.

Bluetooth Low Energy (BLE) is a function of the Bluetooth CoreSpecification Version 4.0 of Bluetooth, which is a short-range wirelesscommunication technology. BLE (also referred to as Bluetooth LE, andmarketed as Bluetooth Smart), is a wireless personal area networktechnology aimed at novel applications in the healthcare, fitness,security, and home entertainment industries, in addition to smart homeand proximity detection services. BLE provides considerably reducedpower consumption and cost compared to classic Bluetooth, whilemaintaining a similar communication range. BLE is natively supported bymobile operating systems, including, for example, iOS®, Android®,Windows® Phone and BlackBerry®, as well as OS X, Linux®, and Windows 8®.BLE involves advantages including, but not limited to, low powerrequirements, operating for “months or years” on a button cell (i.e., asmall single cell battery). As discussed herein, BLE has compatibilitywith a large installed base of mobile phones, tablets and computers.

Additionally, BLE supports “electronic leash” applications, which arewell suited for long battery life, “always-on” devices. As understood bythose of skill in the art, an electronic leash is the pairing(“leashing”) of one or more wireless devices to a host device thatallows the user to find misplaced or out-of-sight objects by activatingthe host device such that the “leashed” object identifies itself. Thus,BLE, via electronic leashing, can allow a single user-operated device tosend communications to a large number of devices or objects.

Turning back to FIG. 1, mobile devices 200-200 n, according to someembodiments, may be implemented in various forms. For example, mobiledevices 200-200 n may include virtually any portable computing devicecapable of connecting to another device and receiving information. Suchdevices include multi-touch and portable devices such as, but notlimited to, cellular phones, smart phones, smart watches, displaypagers, radio frequency (RF) devices, infrared (IR) devices, PersonalDigital Assistants (PDAs), handheld computers, laptop computers,wearable computers, tablet computers, phablets, navigation devices,e-book terminals, integrated devices combining one or more of thepreceding devices, and the like. Mobile devices 200-200 n also mayinclude at least one client application that is configured to receivecontent from another computing device. The client application mayinclude a capability to provide and receive textual content, graphicalcontent, audio content, and the like. The client application may furtherprovide information that identifies itself, including a type,capability, name, and the like. In one embodiment, mobile devices 100may uniquely identify themselves through any of a variety of mechanisms,including a phone number, Mobile Identification Number (MIN), anelectronic serial number (ESN), or other mobile device identifier.

In some embodiments, the mobile devices 200-200 n may communicateidentification information via a BLE communication network. Asunderstood by those of skill in the art, such BLE communication can becoupled to, or implemented via a wireless network. A Wireless networkmay include any of a variety of wireless sub-networks that may furtheroverlay stand-alone ad-hoc networks, and the like, to provide aninfrastructure-oriented connection for mobile devices 200-200 n. Suchsub-networks may include mesh networks, Wireless LAN (WLAN) networks,cellular networks, and the like. Wireless network connectivity mayfurther include an autonomous system of terminals, gateways, routers,and the like connected by wireless radio links, and the like. Theseconnectors may be configured to move freely and randomly and organizethemselves arbitrarily, such that the topology of the wireless networkmay change rapidly. BLE communication between devices 200-200 n mayfurther employ a plurality of access technologies including 2nd (2G),3rd (3G), and/or 4th (4G) generation radio access for cellular systems,WLAN, Wireless Router (WR) mesh, and the like. Access technologies suchas 2G, 3G, 4G and future access networks may enable wide area coveragefor mobile devices, such as mobile device 200 with various degrees ofmobility. For example, a wireless network BLE communication may enable aradio connection through a radio network access such as Global Systemfor Mobile communication (GSM), General Packet Radio Services (GPRS),Enhanced Data GSM Environment (EDGE), Wideband Code Division MultipleAccess (WCDMA), and the like. In essence, such BLE communication mayinclude virtually any wireless communication mechanism by whichinformation may travel between mobile devices 200-200 n.

Such BLE communication between mobile devices 200-200 n at a location100 can be enabled as a stand-alone network to employ any form ofcomputer readable media for communicating information from oneelectronic device to another. For security of the information, suchinformation may be secured by using a known or to be known negotiatedencryption key or a pre-defined encryption key. Such encryption mayoccur at the mobile devices 200-200 n, or some combination thereof.Within the communications networks utilized or understood to beapplicable to the present disclosure, such networks will employ variousprotocols that are used for communication over the network. Signalpackets communicated via a network, such as a network of participatingdigital communication networks, may be compatible with or compliant withone or more protocols.

Signaling formats or protocols employed may include, for example,TCP/IP, UDP, DECnet, NetBEUI, IPX, APPLETALK™, or the like. Versions ofthe Internet Protocol (IP) may include IPv4 or IPv6. The Internet refersto a decentralized global network of networks. The Internet includeslocal area networks (LANs), wide area networks (WANs), wirelessnetworks, or long haul public networks that, for example, allow signalpackets to be communicated between LANs. Signal packets may becommunicated between nodes of a network, such as, for example, to one ormore sites employing a local network address. A signal packet may, forexample, be communicated over the Internet from a user site via anaccess node coupled to the Internet. Likewise, a signal packet may beforwarded via network nodes to a target site coupled to the network viaa network access node, for example. A signal packet communicated via theInternet may, for example, be routed via a path of gateways, servers,etc. that may route the signal packet in accordance with a targetaddress and availability of a network path to the target address.

According to some embodiments, each mobile device 200-200 n maycommunicate or broadcast information directly or indirectly to and fromeach other or among groups of devices. That is, as discussed in moredetail below, mobile devices 200-200 n may communicate informationassociated with identification information, signal-strength informationand/or location information, in addition to orientation informationassociated with the device and a gesture performed via the device.

In accordance with embodiments of the present disclosure, suchcommunication detection is a representation of an ad-hoc network. Thereis, however, no requirement for an exchange of information betweenmobile devices 200-200 n. In accordance with embodiments of the presentdisclosure, mobile devices 200-200 n simply detect communication signals(e.g., broadcast signals) transmitted from each device 200-200 n. Thereis no requirement for the mobile devices 200-200 n to authenticate,connect or exchange data with other devices. Devices 200-200 n, throughsensors in or on the device, only need to detect signals output from theother devices in order to spatially map location 100 by simultaneouslyderiving the direction and distance of the other devices based uponsignal strength measurements as discussed further herein. According tosome embodiments, detection of the signals communicated or broadcastfrom mobile devices 200-200 n may be detected via BLE communication, asdiscussed above.

According to some embodiments, the mobile devices 200-200 n maybroadcast the identification information in a format of an advertisingpacket, as discussed in more detail below. Such communication, orbroadcast, can be implemented via the devices 200-200 n and/or location100 being coupled with an ad server that comprises a server that storesonline advertisements for presentation to users. “Ad serving” refers tomethods used to place online advertisements on websites, inapplications, or other places where users are more likely to see them,such as during an online session or during computing platform use, forexample. Various monetization techniques or models may be used inconnection with sponsored advertising, including advertising associatedwith user. Such sponsored advertising includes monetization techniquesincluding sponsored search advertising, non-sponsored searchadvertising, guaranteed and non-guaranteed delivery advertising, adnetworks/exchanges, ad targeting, ad serving and ad analytics.

For example, a process of buying or selling online or wirelessadvertisements may involve a number of different entities, includingadvertisers, publishers, agencies, networks, or developers. To simplifythis process, organization systems called “ad exchanges” may associateadvertisers or publishers, such as via a platform to facilitate buyingor selling of online advertisement inventory from multiple ad networks.“Ad networks” refers to aggregation of ad space supply from publishers,such as for provision en masse to advertisers. For web portals likeYahoo!®, advertisements may be displayed on web pages resulting from auser-defined search based at least in part upon one or more searchterms. Advertising may be beneficial to users, advertisers or webportals if displayed advertisements are relevant to interests of one ormore users. Thus, a variety of techniques have been developed to inferuser interest, user intent or to subsequently target relevantadvertising to users.

One approach to presenting targeted advertisements includes employingdemographic characteristics (e.g., age, income, sex, occupation, etc.)for predicting user behavior, such as by group. Advertisements may bepresented to users in a targeted audience based at least in part uponpredicted user behavior(s). Another approach includes profile-type adtargeting. In this approach, user profiles specific to a user may begenerated to model user behavior, for example, by tracking a user's paththrough or at a location(s), and compiling a profile based at least inpart on pages or advertisements ultimately delivered. A correlation maybe identified, such as for user purchases, for example. An identifiedcorrelation may be used to target potential purchasers by targetingcontent or advertisements to particular users. During presentation ofadvertisements, a presentation system may collect descriptive contentabout types of advertisements presented to users. A broad range ofdescriptive content may be gathered, including content specific to anadvertising presentation system. Advertising analytics gathered may betransmitted to locations remote to an advertising presentation systemfor storage or for further evaluation. Where advertising analyticstransmittal is not immediately available, gathered advertising analyticsmay be stored by an advertising presentation system until transmittal ofthose advertising analytics becomes available.

FIG. 2 is a schematic diagram illustrating an example embodiment of aclient device that may be used within the present disclosure. Asdiscussed above, a client device can be any type of mobile or portabledevice. For purposes of this disclosure, and for clarity of thefollowing discussion, such devices will be referred to as “mobiledevices”. It should be understood that a mobile device refers to alltypes of portable devices that support BLE, as discussed above. Mobiledevice 200 may include many more or less components than those shown inFIG. 2. However, the components shown are sufficient to disclose anillustrative embodiment for implementing the present disclosure. Thus,mobile device 200 may represent, for example, mobile devices 200-200 ndiscussed above in relation to FIG. 1.

As shown in FIG. 2, mobile device 200 includes a processing unit (CPU)222 in communication with a mass memory 230 via a bus 224. Mobile device200 also includes a power supply 226, one or more network interfaces250, an audio interface 252, a display 254, a keypad 256, an illuminator258, an input/output interface 260, a haptic interface 262, and anoptional global positioning systems (GPS) receiver 264. Power supply 226provides power to Mobile device 200. A rechargeable or non-rechargeablebattery may be used to provide power. The power may also be provided byan external power source, such as an AC adapter or a powered dockingcradle that supplements and/or recharges a battery.

Mobile device 200 may optionally communicate with a base station (notshown), or directly with another computing device. Network interface 250includes circuitry for coupling Mobile device 200 to one or morenetworks, and is constructed for use with one or more communicationprotocols and technologies including, but not limited to, global systemfor Client communication (GSM), code division multiple access (CDMA),time division multiple access (TDMA), user datagram protocol (UDP),transmission control protocol/Internet protocol (TCP/IP), SMS, generalpacket radio service (GPRS), WAP, ultra wide band (UWB), IEEE 802.16Worldwide Interoperability for Microwave Access (WiMax), SIP/RTP, or anyof a variety of other wireless communication protocols. Networkinterface 250 is sometimes known as a transceiver, transceiving device,or network interface card (NIC).

Audio interface 252 is arranged to produce and receive audio signalssuch as the sound of a human voice or audio tones or signals. Forexample, audio interface 252 may be coupled to a speaker and microphone(not shown) to enable telecommunication with others and/or generate anaudio acknowledgement for some action. Display 254 may be a liquidcrystal display (LCD), gas plasma, light emitting diode (LED), or anyother type of display used with a computing device. Display 254 may alsoinclude a touch sensitive screen arranged to receive input from anobject such as a stylus or a digit from a human hand.

Keypad 256 may comprise any input device arranged to receive input froma user. For example, keypad 256 may include a push button numeric dial,or a keyboard. Keypad 256 may also include command buttons that areassociated with selecting and sending images. Illuminator 258 mayprovide a status indication and/or provide light. Illuminator 258 mayremain active for specific periods of time or in response to events. Forexample, when illuminator 258 is active, it may backlight the buttons onkeypad 256 and stay on while the mobile device is powered. Also,illuminator 258 may backlight these buttons in various patterns whenparticular actions are performed, such as dialing another mobile device.Illuminator 258 may also cause light sources positioned within atransparent or translucent case of the mobile device to illuminate inresponse to actions.

Mobile device 200 also comprises input/output interface 260 forcommunicating with external devices, such as a BLE device/unit 100 (fromFIG. 1) or a headset, or other input or output devices not shown in FIG.2. Input/output interface 260 can utilize one or more communicationtechnologies, such as USB, infrared, Bluetooth™, BLE, or the like.Interface 260 may include one or more units for communication betweenthe mobile device 200 and the BLE device 100, or between the mobiledevice 200 and a server. For example, Interface 260 may include a BLEcommunication unit, a mobile communication unit, and a broadcastingreceiving unit. The BLE communication unit supports a BLE communicationfunction. The BLE communication unit may scan the BLE device 100 for apredetermined period of time or upon a request from a user. The BLEcommunication unit may involve a sensor hub. As understood by those ofskill in the art, the sensor hub is a type of Micro Controller Unit(MCU) and may be connected to various types of sensors. The sensor hub,according to some embodiments can collect information about the externalBLE device 100.

The communication unit of interface 260 may also support othershort-range wireless communication functions, in addition to the BLEcommunication function. Short-range wireless technology may include, butis not limited to, a wireless Local Area Network (LAN) which could be aWi-Fi, Bluetooth, WiFi direct (WFD), Near Field Communication (NFC),Ultra WideBand (UWB), or Infrared Data Association (IrDA) network, asdiscussed above with respect to BLE communication. The mobilecommunication unit of interface 260 transmits and receives a wirelesssignal to and from with at least one of a base station, an externalterminal, and a server on a mobile communication network. The wirelesssignals may include, for example, a voice call signal, a video phonecall signal or various forms of data used to transmit and receive textor multimedia messages. The broadcasting receiving unit of interface 260receives broadcasting signals and/or broadcasting-related informationfrom outside, via a broadcasting channel. The broadcasting channel mayinclude, but is not limited to, a satellite channel and a terrestrialbroadcast channel.

Haptic interface 262 is arranged to provide tactile feedback to a userof the mobile device. For example, the haptic interface may be employedto vibrate mobile device 200 in a particular way when the Mobile device200 receives a communication from another user.

Optional GPS transceiver 264 can determine the physical coordinates ofMobile device 200 on the surface of the Earth, which typically outputs alocation as latitude and longitude values. GPS transceiver 264 can alsoemploy other geo-positioning mechanisms, including, but not limited to,triangulation, assisted GPS (AGPS), E-OTD, CI, SAI, ETA, BSS or thelike, to further determine the physical location of Mobile device 200 onthe surface of the Earth. It is understood that under differentconditions, GPS transceiver 264 can determine a physical location withinmillimeters for Mobile device 200; and in other cases, the determinedphysical location may be less precise, such as within a meter orsignificantly greater distances. In one embodiment, however, Mobiledevice 200 may through other components, provide other information thatmay be employed to determine a physical location of the device,including for example, a MAC address, IP address, or the like. Thedevice 200 can also determine the physical location, orientation,coordinates, directional movement or positioning, and the like via, forexample, accelerometers 266, one or more gyroscopes 268, a compass 270,and the like.

Mass memory 230 includes a RAM 232, a ROM 234, and other storage means.Mass memory 230 illustrates another example of computer storage mediafor storage of information such as computer readable instructions, datastructures, program modules or other data. Mass memory 230 stores abasic input/output system (“BIOS”) 240 for controlling low-leveloperation of Mobile device 200. The mass memory also stores an operatingsystem 241 for controlling the operation of Mobile device 200. It willbe appreciated that this component may include a general purposeoperating system such as a version of UNIX, or LINUX™, or a specializedclient communication operating system such as Windows Client™, or theSymbian® operating system. The operating system may include, orinterface with a Java virtual machine module that enables control ofhardware components and/or operating system operations via Javaapplication programs.

Memory 230 further includes one or more data stores, which can beutilized by Mobile device 200 to store, among other things, applications242 and/or other data. For example, data stores may be employed to storeinformation that describes various capabilities of Mobile device 200.The information may then be provided to another device based on any of avariety of events, including being sent as part of a header during acommunication, sent upon request, or the like. At least a portion of thecapability information may also be stored on a disk drive or otherstorage medium (not shown) within Mobile device 300.

Applications 242 may include computer executable instructions which,when executed by Mobile device 200, transmit, receive, and/or otherwiseprocess audio, video, images, and enable telecommunication with anotheruser of another mobile device. Other examples of application programsinclude calendars, browsers, contact managers, task managers,transcoders, database programs, word processing programs, securityapplications, spreadsheet programs, games, search programs, and soforth. Applications 242 may further include messaging client 245 that isconfigured to send, to receive, and/or to otherwise process messagesusing email, SMS, MMS, IM, VOIP, and/or any of a variety of othermessaging communication protocols. Although a single messaging client245 is illustrated it should be clear that multiple messaging clientsmay be employed. For example, one messaging client may be configured tomanage email messages, where another messaging client manages SMSmessages, and yet another messaging client is configured to manageserving advertisements, IMs, or the like.

Having described the components of the general architecture employedwithin the disclosed systems and methods, the components' generaloperation with respect to the disclosed systems and methods will now bedescribed with respect to FIGS. 3-10.

FIG. 3 is a block diagram illustrating the components for performing thesystems and methods discussed herein. FIG. 3 includes a spatial engine300, an associated database 302 for storing spatial tracking informationand an advertisement (“ad”) server 330. The spatial engine 300 could behosted by a mobile device 200-200 n, a separate BLE device(s) atlocation 100, a server, content provider, an ad server, and the like, orany combination thereof. According to embodiments of the presentdisclosure, the spatial engine 300 can be implemented via an installedor web-based application that executes on the mobile device 200.

As discussed in more detail below, spatial context information can bedetected or provided to the spatial engine 300 or accessed by a computerprogram or device that can access the information. Such information, asdiscussed in more detail below, relates to determined distancesassociated between mobile devices at a location, distances betweenmobile devices and other BLE devices (or units) at a location, a mobiledevice's signal strength and the mobile device's identifier, orientationof the device, and the like. Indeed, the spatial information can includea type of mobile device, a user associated with the device, proximitydata or location data associated with the mobile device, number ofmobile devices within a proximity or associated with a location, amongother types of information.

In some embodiments, the spatial information can be stored in a datastructure in storage (e.g., a lookup table and/or other relational dataformat or structure) in database 302, which is associated with alocation, coordinates, or, for example, a commercial entity. Thedatabase 302 can be any type of database or memory that can store theinformation mentioned above. The database 300 can be associated with alocation, a device or a network. That is, for example, spatialinformation associated with a particular location, for example, aperson's house, a mall or stadium, and can be stored in a database 300that is dedicated to such location, as will be evident from the belowdiscussion.

An ad server 330 comprises a server or ad platform that stores onlineadvertisements for presentation to users. As discussed above, “adserving” refers to methods used to place online advertisements onwebsites, in applications, or other places where users are more likelyto see them, such as during an online session or during computingplatform use, for example. Various monetization techniques or models maybe used in connection with sponsored advertising, including advertisingassociated with user. Such sponsored advertising includes monetizationtechniques including sponsored search advertising, non-sponsored searchadvertising, guaranteed and non-guaranteed delivery advertising, adnetworks/exchanges, ad targeting, ad serving and ad analytics.Embodiments of implementations of the ad server 330 in connection withthe spatial engine 300 will be discussed in more detail below.

The spatial engine 300 includes a signal strength module 304, soundsignal module 306, inertia module 308 and a position module 310. Itshould be understood that the engine(s) and modules discussed herein arenon-exhaustive, as additional or fewer engines and/or modules, orsub-engines or modules, may be applicable to the embodiments of thesystems and methods discussed. The operations, configurations andfunctionalities of the engine and each module, and their role withinembodiments of the present disclosure will be discussed in more detailbelow with reference to FIGS. 4-8 and their associated processes.

Turning now to FIG. 4, Process 400 discloses the determination of ad-hocrelative 3D positions of close-by-devices at a location. For purposes ofthis disclosure, “close-by-devices” refers to devices that are at alocation and/or within a predefined distance to one-another. Forexample, 3 users may be at a mall, but only 2 of them are in the samestore—therefore, only the 2 devices in the same store would bedetermined to be “close-by-devices.” However, it should be noted thatthe inclusion or identification of devices as close-by-devices can varydepending on the location, number of devices within a predefined range,signal robustness, or activity/event occurring at the location, amongother factors.

As discussed above, the disclosed systems and methods can determinerelative 3D tracking of devices at a location without additionalinstrumentation than that typically equipped on standard commoditymobile devices, requires no stationary or preconfigured components inthe infrastructure of such devices, and functions with resting as wellas moving devices. In some embodiments, the devices implementing thedisclosed systems and methods must have at least three audio componentsin any configuration and an orientation sensor(s). For example, twospeakers and one microphone, or two microphones and one speaker.According to some embodiments, if at least two devices at a locationsatisfy audio component requirements, all subsequent devices at thelocation only require a single microphone or speaker. Thus, thedisclosed systems and methods can be applied to on any known or to beknown mobile device, as conventional devices feature the requiredminimum audio components.

According to some embodiments, as discussed above and in more detailbelow, the systems and methods discussed herein can be supported bymicrophone-only devices, such as, for example, smart-watches linked to amobile device, where the two devices (watch and phone, or othercombinations of wearable devices such as glasses or one or more items ofintelligent clothing) form one logical unit. For example, once thephone's 3D position is determined, the related watch's location can beinferred despite instances of such devices having fewer audio units(e.g., typically, a smartwatch may only have a microphone, or only onemicrophone and one speaker). As such, treating these two devices as oneconnected logical unit (e.g., the smartphone and smartwatch as discussedabove), the 3D positioning of not only each connected device can bedetermined, but also the 3D positioning of both devices as one logicalunit can be determined.

Process 400 is performed by the spatial engine 300 running on a deviceto detect spatial information about other devices and a location. Asdiscussed above, and in more detail below, implementation of the spatialengine 300 can establish spatial relationships between multiple mobiledevices in order to effectuate ad-hoc file sharing between such devices.Instead of simply selecting a recipient from a list, as withconventional systems, a user is afforded the ability to communicatefiles to another user by simply swiping in the determined direction ofanother device.

The spatial engine 300 performs Process 400 which continuously updatesthe 3D understanding of surrounding devices of a location when one ormore devices changes position. According to some embodiments, thedisclosed systems and methods involve the spatial engine 300 executingat a variety of interactive rates on a mobile device to effectivelycapture the tracking information discussed herein. Such rates can varybased on user input, device manufacture design, application settings,network connectivity environment, and the like.

As discussed in more detail below, the spatial engine 300 achievesrelative 3D tracking of devices at a location by fusing three types ofsignals computed by the signal strength module 304, sound signal module306 and inertia module 308, respectively. A first signal is associatedwith signal strength of BLE signals emitted from each device at alocation. This is processed by the signal strength module 304. A secondsignal is associated with exchanged audio signals between each device.This is processed by the sound signal module 306. The third signal isassociated with the orientation and/or motion signals obtained from eachdevice's sensors, which can be accumulated via dead reckoning. This isprocessed by the inertia module 308. The fusing of such signals iseffectuated by the position module 310, and in some embodiments, theposition module 310 processes the signals via any known or to be knownestimation algorithm or filter, such as but not limited to, an ExtendedKalman Filter (EKF) or other filter for reducing noise and outliers, asdiscussed in more detail below. Therefore, while the discussion belowrecites the use of an EKF, it should not be construed as limiting, asany known or to be known algorithm or filter can be used, as discussedabove.

Each of the above signals (and signal information) contributes to thecalculation of the spatial context information relative to 3D positionsof each device at a location. Process 400 begins with Step 402 where thesignal strength module 304 detects Bluetooth low energy (BLE)information from which information about the presence of surroundingdevices and their identifiers can be determined. As discussed in moredetail below in connection with FIG. 5, the signal strength module 304can estimate the coarse distance to each peer device at a location fromthe strength of this signal. According to embodiments of the presentdisclosure, this information is continuously updated to reflect acompilation of present devices at a location and their coarse distancesto each other, thereby establishing reliable spatial tracking.

According to some embodiments, when a predetermined threshold number ofdevices are present at a location, for example four devices, the signalstrength module 304 can derive addition 3D location information byinterpolating signal strengths between all present devices. According tosome embodiments, the spatial engine 300 can leverage any known or to beknown existing indoor tracking and/or navigation system (e.g., iBeacon®)based on such BLE information derived by the signal strength module 304to further refine and stabilize calculated 3D positions.

In Step 404, the sound signal module 306 detects acoustic (or sound oraudio) signals and their associated information are utilized to infer 3Dpositions, as discussed in more detail below in connection with FIG. 6.According to some embodiments, each device at a location repeatedlycommunicates (or plays) acoustic signals that provide each device'sidentifier. From this, the sound signal module 306 can also determinearrival timestamps of all signals, which can then be communicated orbroadcasted to the other devices at the location. For example, devicescan communicate audio signals via a speaker of the device and receive orrecord signals from other surrounding devices via a microphone on thedevice. From the differences in arrival timestamps on all participatingdevices at a location, the sound signal module 306 computes the distancebetween devices. According to some embodiments, the distance can becomputed to the centimeter or another larger or smaller accuracymeasurement, which may vary over time or iteration of timestampbroadcast as more data is collected and confidence in the data isincreased.

Therefore, the spatial engine 300 can leverage the presence of devicesto calculate relative 3D positions of devices from the set of distancesbetween speaker-microphone combinations between devices. It should benoted that embodiments of the systems and methods discussed hereinperform such calculations without synchronizing devices' audio clocks.Also, according to embodiments of the present disclosure, the exchangedsound (or acoustic) signals can be audible (chirps, bursts or fixed ormulti-frequency tones, for example) or inaudible—for example, between19-22 kHz, or some other fixed, variable or multi-frequency that canvary depending on factors present at the location or with the devicesbroadcasting such signals.

Additionally, BLE-based distance estimations from Step 402 can becalibrated continuously and in near-real time (i.e., on the fly) byusing the distances that result from exchanged audio signals in Step404, as discussed in more detail below. Since BLE signal strengths arefairly invariable for a given pair of devices, distance and environment,the signal strength module 304 can repeatedly update determined BLEsignal strengths to accurate distances. This not only refines theaccuracy of the distance cues between devices, but substantiallyimproves the suitability of BLE signals for distance tracking in thecase that audio signals become too unreliable.

In Step 406, the inertia module 308 detects information (or values) fromdevice motion and orientation data produced from inertial sensors ineach device. As discussed in more detail below in connection with FIG.7, the inertia module 308 can implement, for example, any known or to beknown dead reckoning filter or algorithm to compile the motion andorientation information/values and detect device motions and gestureswith low latency. According to some embodiments, while the data frominertial sensors is naturally subject to drift, the inertia module 308can use the BLE and/or acoustic signals to offset estimates from theinertia sensors, which provide much higher accuracy.

In Step 408, as discussed in more detail below in connection with FIG.8, the position module 310 integrates the information determined by thesignal strength module 306, sound signal module 308 and inertia module310 by, by way of one non-limiting example, by fusing them through anEKF to determine relative 3D positions of each device at the location.The position module 310 can then then broadcast the 3D positioninginformation to the other devices at the location. Step 410. Step 410involves communicating or enabling access to a 3D spatial mapping of thelocation which identifies each device's position in a 3D space and anidentifier of each device. According to some embodiments, a user canselect another device and communicate information, such as content,media files and the like, to the selected device, where suchcommunication can occur over a network associated with the location (asdiscussed above), or via a BLE packet (as discussed below).

Turning now to FIGS. 5-8, each Figure will now be discussed in order todetail the steps being performed in Steps 402-408, respectively. Thatis, FIG. 5 discusses the details of Step 402 performed by the signalstrength module 304. FIG. 6 discusses the details of Step 404 performedby the sound signal module 306. FIG. 7 discusses the details of Step 406performed by the inertia module 308. FIG. 8 discusses the details ofStep 408 performed by the position module 310.

FIG. 5 begins with Step 502 where the signal strength module 304continuously scans a location for available devices with an active BLEsignal. That is, a location is monitored to identify other devices atlocation within a threshold (or predefined) range that are emitting aBLE signal. In some embodiments, the strength of each BLE signals isalso identified. In some embodiments, the monitoring or scanning occursat a predetermined rate, for example, between 5 Hz and 10 Hz, or atfixed or variable time intervals such as every 500 ms or every twoseconds; however, it should not be construed to limit the presentdisclosure to such range as the range can vary depending on signalstrength, location environmental factors, user settings, devicesettings, application settings, and the like. As a result of thescanning, a list of detected devices is compiled and stored in adatabase (or lookup table), where the list identifies each device'sidentifier (ID). Step 504.

For each detected device, the signal strength module 304 determines aninitial BLE signal-strength measurement between devices. Step 506. Thisstep is based on a derived rough or estimated distance between devicesat the location. Step 506. The estimated distance information can alsobe stored in the lookup table in association with the appropriatedevice's ID. In some embodiments, the rough distance is resultant asimple mapping from the signal strengths of the detected BLE signals. Byway of a non-limiting example, the estimate can include a table mappingof signal strengths −45 to −80 dB in increments of 5 dB distances.

As an example for Steps 502-506, device 1 monitors a location anddetects device 2 and device 3 emitting BLE signals. A listing iscompiled and stored in a database which includes the device identifiersfor devices 2 and 3. A rough or estimated distance is determined betweendevice 1 to device 2, and device 1 to device 3 based on the detected BLEsignals, which is also stored in the database.

In Step 508, the signal strength for each device is refined. Asdiscussed in more detail below, the refinement of such mapping is basedon a distance measurement resultant from the sound/audio informationdetermination performed in Step 404 (and detailed below in FIG. 6). Thatis, Step 508 involves mapping the rough/estimated signal strengths(which are compiled and stored in Step 506) to the audio distancesdetermined by the signal sound module 306 (which is discussed above inStep 404 and in more detail below in FIG. 6). The refined signalstrength for each device is stored in the database (lookup table orrelational database or other data structure), and according to someembodiments, the spatial engine 300 can interpolate all subsequent BLEbased distance estimates determined by the signal strength module 304using such table. This provides a real-time (or “on-the-fly”)calibration increasing the accuracy of the spatial engine 300'ssignal-strength-to-distance mapping by leveraging the robustness of BLEsignals.

FIG. 6 discloses the details of Step 404 performed by the sound signalmodule 306 for determining audio-distance measurements between thedevices at the location. FIG. 6 begins with Step 602 which includesdetermining audio signal playback information. Step 602 involvessonifying a sender ID and speaker ID based on a determined distancebetween devices from a time-of-flight analysis of exchanged soundsignals. This involves determining a precise moment (within designparameters and device capabilities, which, it is expected, will improveover time) or time interval at (or over) which sound signals reach (orare received at) each device.

By way of background, audio signals typically comprise two parts: aninitial chirp or short burst to announce the signal and an ensuingpayload. Audio signals can be broken down into a number of frames persecond (or millisecond, for example) for communication of the signal tobe received as a continuous stream of information, as opposed toreceiving the signal as a singular packet. According to embodiments ofthe present disclosure, the audio signals discussed herein furtherinclude a sender ID, speaker ID and a counter encoded in the payload.

The purpose of the initial chirp in each audio signal is for thereceiver to determine the precise frame at which the signal has beenreceived. For this reason, the sound signal module 306 can use a linearup/down chirp (i.e., a sweep signal), which exhibits lowcross-correlation and high auto-correlation, therefore facilitatingprecise detection. In addition, such chirps are robust to device motion,noise, and ambient sounds. The up chirp utilized in the example hereinspans the entire frequency range available to spatial engine 300 (forexample, 19-22 kHz—the inaudible range). The duration of frames in theaudio signal is long enough for a sufficiently loud or robust sound toform, yet short enough to avoid influence from sounds from other sourcessuch as ambient noise or voices.

As discussed above, sound signal module 306 encodes the sender's ID(e.g., device ID of the device emitting the acoustic signal), thespeaker ID of the device and a counter into the payload of a signal.This enables receivers of the signal to determine the origin of eachsignal. According to some embodiments, predetermined portions (or bits)of the payload can be allocated for each of the encoded information. Forexample, 5 bits for encoding the sender ID can be allocated in thepayload. In some embodiments, the sender ID can result from a hash ofthe device's BLE identifier. In another example, a 3-bit counter can beencoded into the payload of each message, which ensures that the arrivaltime of the signal may be compared across devices.

According to some embodiments, a key portion of the payload is theencoding using Hamming codes. This involves encoding concatenated bitsinto single bit units using two Hamming codes. For example,concatenating 5+3 bits into 8 bits, using two (4,7) Hamming codes. Insome embodiments, Hamming codes improve the reliability of correctsignal transmission as well as enabling the detection of overlappingsignals, but other coding schemes now known or to be known can beutilized.

The sound signal module 306 can sonify (e.g., use of non-speech audio toconvey information or perceptualize data) using biorthogonal chirps(also referred to as “candidate” signals). Such chirps can represent apredetermined number of bits (e.g., 2 bits) as a series of up or downchirps (e.g., 4 up/down chirps), whereby each chirp is orthogonal to theother chirp(s). This enables the recognition of each chirp series to berobust in light of other superimposed chirps. According to someembodiments, in order to avoid clicking or cracking of speakers whenplaying biorthogonal chirps in the inaudible spectrum, the signal soundmodule 306 can implement a fade-in and fade-out area of a predeterminednumber of frames at the beginning and/or end of a chirp. As such, forexample, the fade-out frames of a prior chirp can be superimposed on thefade-in frames of the following chirp. It should be understood thatwhile the discussion above involves biorthogonal chirps, it should notviewed as limiting, as the discussion above in connection with Step 602can be effectuated with any other known or to be known types of soundencoding, including amplitude, frequency, chirp rate and code divisionmultiple access (CDMA) encoding, and the like.

Step 604 involves detecting and decoding recorded audio signals.According to some embodiments, the disclosed systems and methods arebased on the frame counter of the audio input stream, which encodes thetimestamp running along a constant time rate. As discussed above, thespatial engine 300 does not require synchronization across input andoutput channels, as output from each device runs independently and mayoccur at arbitrary times. Note: the frame count will be referred to asthe timestamp in the remainder.

According to some embodiments, in order to detect the initial chirp ofan audio signal, the sound signal module 306 continuously analyzes theinput stream from the microphone of another device at the location. Amatched filter is run on the high-passed input signal in order tocorrelate the signal with the up chirp discussed above. Then, the soundsignal module 306 searches the peak correlation value through a peakdetection algorithm, and stores the frame count as a candidate signal.Such correlation results from the similarity with the up chirp and itsamplitude in the recorded stream. Thus, Step 604 includes decoding allcandidate signals. More specifically, the biorthogonal chirps thatsonify the payload starting at the offset after each candidate frame aredecoded.

Step 604 further includes decoding the biorthogonal chirps bycorrelating each part of the audio stream with each possibleallocation(s) of a biorthogonal chirp. For each part of the chirp, asample with the highest correlation value is selected. This processresults in a predetermined number bits, which represents the Hammingcode of the candidate signal. This coding is then decoded to obtain anumber of bits, as well as an indicator that provides informationindicating whether the Hamming code applied above is flawless (forexample, an 8 bit decoded value can signify flawless audio information,where the number of bits can be a predetermined value, or depend uponthe type or value of the candidate signals/chirp). If an error isdetected, this candidate is discarded. Additionally, in someembodiments, the result of this detection Step 604 is discarded if thechirped sender ID is mismatched.

According to some embodiments, if the detection step 604 results inmultiple flawless decoded candidate results, the sender ID and speakerID of each decoded signal are compared and the earliest in time signalis selected; and in some embodiments, the later in time signal isdiscarded. Thus, Step 604 results in producing reliable decoding ofsuperimposed sounds originating from different devices.

Step 606 includes broadcasting arrival timestamps to other devices atthe location. More specifically, once a device has determined theprecise frame at which a signal has been received, the sound signalmodule 306 transmits this timestamp to surrounding devices along withthe receiver's device ID, device model, device orientation, and thedecoded values as shown below in an example of a “receiver packet.”

receiver ID + receiver's receiver's 3D decoded decoded decoded modeltimestamp orientation sender ID speaker counter

Depicted above is an example of a receiver packet that can be radioed orotherwise transmitted and received over an ad-hoc wireless connection tosurrounding devices upon detecting and decoding a received audio signal.According to some embodiments, instead of a receiver packet, a BLEpacket or BLE advertising packet may be used or reused to communicatesuch information. As such, utilizing the BLE packet(s), the informationcould be broadcast without the ad-hoc network as it could be broadcastusing BLE technology, as discussed above in connection with FIG. 1.

In addition to broadcasting the receiver packet after decoding a signal,the sound signal module 306 responds with a regular audio signal. Thisensures that each participating device that has emitted an audio signalwill hear back from surrounding devices. That is, every device willrecord a timestamp for an audio signal emitted by every device—includingfrom itself. Step 606 results in stored timestamps from both, decodedaudio signals as well as incoming receiver packets locally. According tosome embodiments, the following distance calculations of Step 608 areinvoked whenever new timestamps are broadcast (and received by adevice), where the communication can occur through audio/radio or BLEcommunications.

Step 608 includes computing speaker-microphone distances fromtimestamps. That is, using an example of two devices: Device A andDevice B for clarity, the distance between two devices is calculatedbased on the timestamps of two audio signals emitted by one of the twodevices. This produces four timestamps overall: for example, Device A'ssignal as observed by itself (t₀) and by Device B (t₁), and Device B'ssignal by itself (t₂) and by Device A (t₃). According to embodiments ofthe present disclosure, these four timestamps produce the sum of thedistances from the speaker on Device A to the microphone on Device B andfrom the speaker on Device B to the microphone on Device A. The sum ofthese distances can be obtained:

d _(sum) =c((t ₃ −t ₀)−(t ₂ −t ₁))+d _(A) +d _(B),  (Eq. 1)

Here c denotes the speed of sound under the current conditions at thelocation and d_(A) and d_(B) denote the distances between the speakerand microphone on Device A and Device B, respectively. Eq. 1 evidencesthat there is no requirement for calibration of audio clocks betweendevices, for example: Device A and Device B. This computation isperformed for each combination of speaker to microphone.

According to some embodiments, the sums of distances between all thedevices at a location can be used two-fold. First, half of the averageof all the distance sums can be fed back to update/refine the BLE signalstrength-to-distance mapping, as discussed above in connection with Step508. As discussed above, this improves the BLE signal information'sstatic initialization mapping. Secondly, the distance sums betweendevices are utilized to map, compute and track the relative 3D positionsof the devices at the location. That is, the position of Device A andthe distance between Device A and B are known; thus, the 3D positioningof Device B to Device A can be determined, as discussed in more detailbelow.

FIG. 7 discloses further details of Step 406 performed by the inertiamodule 308 for determining the input from the inertial sensors of thedevices at the location. FIG. 7 begins with Step 702 which includes theinertia module 308 continuously monitoring the location to identify anddetermine orientation information of a device based on the device's 3Dorientation derived from the inertia sensors of the device. In Step 704,the device shares the orientation information with other devices uponreceiving an audio signal as described above in Step 604. According tosome embodiments, monitoring of the inertial sensors of a device by theinertia module 308 enables the detection and tracking of 3D devicemotions as well as gestures that are useful for interaction betweendevices, as discussed above.

According to some embodiments, to obtain stable device orientation, thedevice rotation can be anchored by a magnetometer, which can either belocated at the location, or in some embodiments, on at least one deviceat the location. The magnetometer creates a rotation reference acrossall devices. In order to compensate for situations of inaccurate and/orslow magnetometer readings, yaw rotations from the gyroscope values canbe determined, whose drift can be corrected using the accelerometer ofthe device when the device is not in motion. For example, locationmotions based on inertial sensors of a device can be detected based ondetermined patterns of left motions, right motions, up motions and/ordown motions associated with the device. As a result, Step 406 resultsin the detection of reliable device rotation information and orientationinformation with low latency.

According to some embodiments, the inertia module 308 can additionallyimplement dead reckoning using inertial sensors of the device to provideresponsive tracking. Specifically, the inertia module 308 can detect ifa device is in motion or stationary using a mechanical filter and/orsmoothing window, for example. That is, when set in motion, the inertiamodule 308 can detect the 3D direction through pattern matching andestimate the moved distance based on peak acceleration and time.According to some embodiments, the detection of speed and/or directionchanges may only be used to fill in gaps between two subsequent audiosignals, whereby such gap filling enables a more accurate 3D positionsas described below.

FIG. 8 discloses further details of Step 408 performed by the positionmodule 310 which fuses the signal information determined in Steps 402,404 and 406, as discussed above in relation to FIGS. 4-7. The positionfilter 310 obtains (or receives) the determined signal information fromthe signal strength module 304, the sound signal module 306 and theinertia module 308, and executes an EKF algorithm on the signalsassociated with each device at a location. Step 802.

According to embodiments of the present disclosure, the modules of thespatial engine 300 execute at different rates on a device. By way ofnon-limiting example, the signal strength module 304 for determining BLEsignal strength-to-distance information can execute at a frequency of 5Hz; the sound signal module 306 for determining audio-based 3Dpositioning information can execute at 5 Hz; and the inertia module 308,which can represent an inertia measurement unit (IMU) for a device, canexecute at 60 Hz.

Thus, in Step 804, the inertia module 310 can execute an EKF to fuse thesignal information and smooth the measurements from the differentmodules. According to some embodiments, the EKF can execute recursivelyto compensate for minor or unintentional movements or rotations of adevice detected by the inertia module 308 (from Step 406), and tocompensate for noise, physical obstructions, ultrasound interference,and the like (from Step 404). As discussed above in Step 410, afterexecuting the EKF algorithm (via the EKF filter) on the aggregate signalinformation, the filtered information is then broadcast to the otherdevices at the location, where the broadcasted information comprises therelative 3D positioning of the devices at the location.

It should be understood that while the instant disclosure involves theapplication of an EKF, it is not so limiting, as any other known or tobe known sensor (or signal), data and/or information fusion andintegration algorithm or filter can also be utilized in a similarmanner, as discussed herein, and not diverge from the scope of thepresent disclosure.

According to some embodiments, the EKF filter, while processing theaggregate signals of the spatial engine 300, can implement a statetransition and control model, and an observation model to perform the 3Dspatial measurements discussed herein. This ensures that the signalstrength from the BLE measurements, distance determinations from theaudio measurements and positioning determinations from the inertiameasurements are accurate and up-to-date.

According to some embodiments, the position module 310 determines thelocal coordination system of the location. That is, since allsurrounding device positions that are calculated are relative to thedevice making the calculations, a local coordination system is definedto ensure that the 3D positioning is consistent with the orientation anddirectional determinations of the calculating device. In other words,for each device that runs implements the spatial engine 300, the devicedefines itself as the center in the coordinate system; then, therelative coordinates of surrounding devices at the locations are definedas a state model by the EKF. As illustrated in FIGS. 12-13, thecoordinates and orientation of a device 1200 can be determined. Forexample, as illustrated in FIG. 12, device 1200 comprises a microphone1202 and speaker 1204. As discussed above, device 1200 can include avariety of microphones and speakers and the illustration of device 1200only displaying one microphone 1202 and one speaker 1204 should not beconstrued as limiting as each component can indicate a plurality of suchcomponents. Device 1200 is shown with respect to an x-axis and y-axis.As illustrated in FIG. 12, the line connecting the microphone 1202 andspeaker 1204 defines the y-axis, where the x-axis is orthogonal to they-axis and the z-axis is the normalized axis to the x-y axes. Indeed, asillustrated in FIG. 13, device 1200's x-y-z axes are illustrated inaccordance with the above discussion. FIG. 13 further illustrates a“+roll” and “−roll” which shows that device 1200 can change orientationand movement positions as detected by the gyroscope, compass oraccelerometer(s) associated with device 1200, as discussed above and inmore detail below. Device 1200's “y_(dw)” indicates a rotational valueassociated with device 1200's orientation and position, as determined bythe gyroscope, compass or accelerometer(s) associated with device 1200,as discussed above and in more detail below.

FIG. 13 also shows that the microphone(s) 1202 and speaker(s) 1204 ofdevice 1200 can detect audio signals in any direction, which can bereferred to as “+pitch” and “−pitch” based on the directional axis thesignal is communicated.

According to some embodiments, the x-axis and y-axis are definedaccording to the orientation of the device. In some embodiments,coordinates' axes can be defined according to the coordinates of thelocation, another device, and device 1200, or some combination thereof,as discussed herein.

For example, for a device having coordinates Px, Py, Pz (representingthe x-, y-, z-axis coordinates at a location as determined in accordancewith the above discussion in relation to FIGS. 12-13) at time T, thefollowing formula is used to determine a positional transitional model(or coordination system):

S _(k) =[P _(x) ,P _(y) ,P _(z)]^(T),  (Eq. 2)

According to some embodiments, a control model is determined. That is,from Step 406's dead reckoning estimations, the position module 310 canaccount for quick and/or minor device motions using the IMU (from theinertia module 308). These inferences are utilized as input to the aboveequation to produce a control model for the location:

u _(k) =[Δx,Δy,Δz] ^(T),  (Eq. 3)

Using the above state model from Eq. 2 and the control model from Eq. 3,an updated state model is formed. That is, for example, using twodevices at a location: Device A and Device B, the state model can beformed as:

S _(k) =A·S _(k-1) +B·u _(k-1) +w _(k),  (Eq. 4),

Where A signifies the signal information associated with Device A, and Bsignifies the signal information associated with Device B. W_(k)signifies an offset associated with a device (such as from the deadreckoning), as discussed above, which accounts for discrepancies betweenthe two devices' readings. Indeed, the offset accounting ensures thatthe raw data accumulated by the devices is analyzed to remove anyunreliability from any unintentional movements of the device and/ornoise detected with each signal.

According to some embodiments, the EKF filter can also establish anobservation model that is specific to the location. As discussed above,by way of an example for explanation purposes using a location havingtwo devices present, four distinct distances are determined between thedevices based on the exchanged audio signals (Step 404) and anotherdistance determination based on BLE signals (Step 402). An observationmodel can be established based on these determined distances. The fouraudio-based distances, referred to as (d_(ll), d_(lr), d_(rl), d_(rr))are processed to get an initial observation:

Z _(k) =[d, cos θ, cos φ]^(T),  (Eq. 5)

Where d represents each audio-based distance, and θ is the angle betweena vector pointing to another device and reference device's y-axis. φ isa similar angle on the other device. For example, between two of thedevices, Devices A and B among the four devices:

$\begin{matrix}{{{\cos \; \theta} = \frac{{A_{r}E_{m}} - {A_{l}E_{m}}}{W}},{{\cos \; \phi} = \frac{{E_{r}A_{m}} - {E_{l}A_{m}}}{W}},} & \left( {{Eq}.\mspace{14mu} 6} \right)\end{matrix}$

For the distance determined from BLE determinations, the d value isupdated due to changes in: cos θ, cos φ, which are typically relativelysmall during the short distance movement. Thus, Device A and Device B'srelative positions can be inferred in connection with each device'sy-axis:

$\begin{matrix}{{{M_{A}\left( {a_{1},b_{1},c_{1}} \right)}\mspace{31mu} {S_{AL}\left( {a_{2},b_{2},c_{2}} \right)}\mspace{31mu} {S_{AR}\left( {a_{3},b_{3},c_{3}} \right)}},} & \left( {{Eq}.\mspace{14mu} 7} \right) \\{{{M_{B}\left( {a_{4},b_{4},c_{4}} \right)}\mspace{31mu} {S_{BL}\left( {a_{5},b_{5},c_{5}} \right)}\mspace{31mu} {S_{BR}\left( {a_{6},b_{6},c_{6}} \right)}},} & \left( {{Eq}.\mspace{14mu} 8} \right)\end{matrix}$

According to some embodiments, the EKF filter can also determine arotation matrix R from the IMU measurements of the inertia module 308,which maps a device's coordinates to polar coordinates. Based on this, adevice's relative coordinates can be translated to the referencedevice's local coordination system. For example, M_(B), for Device B,has a relative coordination to Device A's coordination system, whichresults in:

$\begin{matrix}{{M_{R}^{T} = {R_{A}^{- 1}R_{B}M_{B}}},} & {\left( {{Eq}.\mspace{14mu} 9} \right).}\end{matrix}$

Similarly, the EKF can denote

$\begin{matrix}{{{M_{2}^{T}\left( {a_{4}^{T},b_{4}^{T},c_{4}^{T}} \right)}\mspace{31mu} {S_{1L}^{T}\left( {a_{4}^{T},b_{4}^{T},c_{4}^{T}} \right)}\mspace{31mu} {S_{1R}^{T}\left( {a_{6}^{T},b_{6}^{T},c_{6}^{T}} \right)}},} & {\left( {{Eq}.\mspace{14mu} 10} \right).}\end{matrix}$

Therefore, given any positional state [P_(x),P_(y),P_(z)] and tworotation matrixes, the distance between devices (e.g., from a microphoneof Device A to a speaker of Device B) can be inferred as follows:

$\begin{matrix}{{{A_{l}B_{m}} = \sqrt{\left\{ {\left( {a_{1}^{T} + x - a_{2}} \right)^{2} + \left( {b_{1}^{T} + y - b_{2}} \right)^{2} + \left( {c_{1}^{T} + z - c_{1}} \right)^{2}} \right\}}}\mspace{79mu} {{Z_{k} = {{h\left( S_{k} \right)} + {H \cdot \left( {S_{k} - S_{k}^{-}} \right)} + v_{k}}},}} & \left( {{Eq}.\mspace{14mu} 11} \right)\end{matrix}$

where h can be inferred from Eq. 6, and H is a Jacobian matrix of h.Thus, as discussed above, EKF algorithm can be implemented torecursively execute so as to account for time and measurement updatesfor each device at a location.

According to some embodiments, the disclosed systems and methods canemploy a residual model to calculate the 3D offset, as discussed above.In some embodiments, all determinable points in a location's space aresearched in order to calculate a residual error of each point—each pointhaving x, y and z coordinates within the location. In some embodiments,each determinable point may only involve points having a device'slocation associated therewith. In some embodiments, such searching andcalculation can be performed by any known or to be known residualmodelling technique and/or algorithm. The point identified as having thesmallest error is identified, and based on this point, the residualcalculation for the entire location can be determined. That is, from theinitial observation discussed above, in reference to the fouraudio-based distances referred to as (dll, dlr, drl, drr), the residualerror for any position (x, y, z) (e.g., any device's position in thelocation) can be calculated based on a simulation of another 4audio-based distances based on orientation data of a device (orientationdata discussed above), thereby resulting the data set: (simu_(dll),simu_(drl), simu_(dlr), simu_(drr)). Therefore, the calculated residualerror for a position (x, y, z) is determined as follows:

(x,y,z)=[(simu_(dll) −dll)²+(simu_(dlr) −dlr)²+(simu_(drl)−drl)²+(simu_(drr) −drr)²]   (Eq. 12).

In accordance with embodiments of the present disclosure, in connectionwith the above discussion related to FIGS. 3-8, below is a non-limitingexample described to illustrate the spatial engine 300's extension to 3Dspatial mapping and tracking for cross-device interactions.

For example, there are over 50 people present at a restaurant; however,only 3 people are seated together at a table, therefore these threepeople, and their associated mobile devices are determined to closelycollocated. The three users have devices: Device A, Device B and DeviceC, respectively. Applying the systems and methods discussed herein, thespatial determination process involves determining a signalstrength-to-distance measurement based on the BLE signals emitting fromeach person's device. For purposes of this example, Device A is thereference device. Therefore, Device A identifies that Device's B and Care emitting BLE signals and determines the signal strength of eachdevice's BLE emission.

Device A then performs the audio-based distance determination based oninaudible acoustic signals communicated to and from Device A to DevicesB and C. As discussed above, this involves Device A communicating audiosignals via Device A's speaker to Device B and C for reception by eachdevice's microphone. In turn, Device B and C communicate return signalsvia their speakers for reception by Device A's microphone. Additionally,each device broadcasts the timestamps of each audio transmission. Asdiscussed above, the timestamp transmission can be communicated via areceiver packet as another audio transmission, or communicated via BLEsignals between devices.

Additionally, inertial readings of each device are performed andcommunicated between the devices. The inertial readings providemeasurements (i.e., IMU) associated with each device's orientation andmovements at/around the location.

After calculating the BLE measurements, audio-distance measurement andIMU, which can performed in any order as the present disclosure is notlimited to any specific order of calculation, each measurement is fed toa filter running a Kalman algorithm, referred to as EKF as discussedabove. The EKF aggregates and fuses this information together to producea 3D tracking of the location. As a result, a relative 3D mapping of thelocation, which reveals the locations of the devices in the room, iscommunicated to each device.

According to some embodiments, 3D mapping of all the devices at alocation within a structure can be provided to each device at thelocation. The 3D mapping can be a rendering of the location whichincludes or shows the devices that are present at a location (e.g., aroom), and in some embodiments each device's ID (or associated user ID).The rendering can also indicate the position of each device in thelocation. The positioning can be provided by displaying the surroundingdevices in the 3D mapping in accordance with the devices' direction andposition, where such positioning can be relative to a particular deviceor based on the coordinates of the MOM.

For example, as illustrated in FIG. 11, location 1100 can be a store ina mall where 3 devices are located. That is, devices 1102, 1104 and 1106are concurrently located at the location. This is similarly displayed inFIG. 1 as discussed above. Using device 1102 as the reference device,device 1102 can display a user interface 1110 that displays the 3Dmapping 1108 of the room 1100. The dashed lines in the figureillustrated that the devices can communicate with each other over abroadcast network, via BLE packets, via radio signals, over a wirelessnetwork, via NFC communication, or any other communication technology,as discussed above. Indeed, as discussed above, information communicatedbetween each device could be broadcast without an ad-hoc network as itcould be broadcast using BLE technology, as discussed above inconnection with FIG. 1.

Device 1102 illustrates a display of the 3D mapping 1108, which showsthe positioning of device 1104 and device 1106 respective to device1102's position in the room 1100. As illustrated in display 1108, device1104 is located in the northeast direction of the room (or to the upperright hand corner of the room respective to device 1102's location inthe room) and device 1104 is located in the southeast direction of theroom (or to the lower right hand corner of the room respective to device1102's location in the room). As discussed above, based on suchrendering of the 3D mapping 1108, device 1102 can communicate withdevice 1104 and 1106. For example, device 1102 can share files withdevice 1104 through the user of device 1102 swiping the file (via atouch screen interface of device 1102 as understood by those of skill inthe art) in the direction of device 1104. As discussed above, thesharing of the file can be communicated via NFC or BLE packettransmissions, in addition to any other known or to be knowncommunication protocol, as discussed above.

According to some embodiments, the 3D mapping relative to thedetermining device; therefore, the 3D mapping of the devices in theroom, according to this example, are respective Device A's position inthe room. This 3D spatial mapping can be used to media exchange, andeven for monetization purposes, as discussed below.

FIG. 9 is an example of a work flow 900 of serving advertisements todevices based on the 3D spatial mapping discussed above in FIGS. 3-8.Specifically, FIG. 9 illustrates example embodiments of howadvertisements are served to users of mobile devices. In Step 902, thedetermined location information of a user's device at a location isidentified. This identification is derived from the 3D spatial mappingof the location from Steps 402-410. In Step 904, the locationinformation is communicated (or shared) to an advertisement platformcomprising an ad server and ad database. Upon receipt of the locationinformation, the advertisement server performs a search of the addatabase for a relevant advertisement. The search for an advertisementis based at least on the location information. Specifically, Step 904involves the advertisement server searching the advertisement databasefor an advertisement(s) that is associated with or matches theidentified location within the location. For example, if a user is in amall, and is determined to be located near a restaurant within the mall,the selected ad can correspond to a coupon associated with thatparticular restaurant. This determined location is based, at least inpart, on the 3D spatial mapping of the location, as discussed above.

FIG. 9 also illustrates an embodiment where content that is being sharedbetween users at the location can be used to determine advertisementinformation. For example, after determining the 3D spatial mapping of alocation, a user may perform a media exchange with another user at thatlocation, where that media can be used as the context for searching anad platform. In another example, if two users are determined to bewithin a predetermined distance to each other based the 3D mapping(e.g., they are next to each other most likely viewing content on one ofthe user's device), a context can be determined based on the contentbeing viewed/rendered. Thus, Step 901 includes determining a contextfrom content being rendered by at least one device at the location. Asdiscussed above, such rendering can include, but is not limited to,playing or viewing content, transferring content, storing content,uploading content, downloading content, and the like. Step 903 issimilar to Step 904, in that an ad platform is searched, however, inStep 903 the ad database is searched for an ad(s) associated with thedetermined context from Step 901.

In Step 906, an advertisement is selected (or retrieved) based on theresults of Step 903 and/or 904. In some embodiments, the advertisementcan be selected based upon the result of Steps 903 and/or 904, andmodified to conform to attributes of the device upon which theadvertisement will be displayed. In Step 908, the selected advertisementis shared or communicated to at least one user's device at the location.In some embodiments, the selected advertisement is sent directly to eachuser's mobile device through applicable communication protocol and/orcommunication application.

As shown in FIG. 10, internal architecture 1000 includes one or moreprocessing units, processors, or processing cores, (also referred toherein as CPUs) 1012, which interface with at least one computer bus1002. Also interfacing with computer bus 1002 are computer-readablemedium, or media, 1006, network interface 1014, memory 1004, e.g.,random access memory (RAM), run-time transient memory, read only memory(ROM), media disk drive interface 1020 as an interface for a drive thatcan read and/or write to media including removable media such as floppy,CD-ROM, DVD, media, display interface 1010 as interface for a monitor orother display device, keyboard interface 1016 as interface for akeyboard, pointing device interface 1018 as an interface for a mouse orother pointing device, and miscellaneous other interfaces not shownindividually, such as parallel and serial port interfaces and auniversal serial bus (USB) interface.

Memory 1004 interfaces with computer bus 1002 so as to provideinformation stored in memory 1004 to CPU 1012 during execution ofsoftware programs such as an operating system, application programs,device drivers, and software modules that comprise program code, and/orcomputer executable process steps, incorporating functionality describedherein, e.g., one or more of process flows described herein. CPU 1012first loads computer executable process steps from storage, e.g., memory1004, computer readable storage medium/media 1006, removable mediadrive, and/or other storage device. CPU 1012 can then execute the storedprocess steps in order to execute the loaded computer-executable processsteps. Stored data, e.g., data stored by a storage device, can beaccessed by CPU 1012 during the execution of computer-executable processsteps.

Persistent storage, e.g., medium/media 1006, can be used to store anoperating system and one or more application programs. Persistentstorage can also be used to store device drivers, such as one or more ofa digital camera driver, monitor driver, printer driver, scanner driver,or other device drivers, web pages, content files, playlists and otherfiles. Persistent storage can further include program modules and datafiles used to implement one or more embodiments of the presentdisclosure, e.g., listing selection module(s), targeting informationcollection module(s), and listing notification module(s), thefunctionality and use of which in the implementation of the presentdisclosure are discussed in detail herein.

Network link 1028 typically provides information communication usingtransmission media through one or more networks to other devices thatuse or process the information. For example, network link 1028 mayprovide a connection through local network 1024 to a host computer 1026or to equipment operated by a Network or Internet Service Provider (ISP)1030. ISP equipment in turn provides data communication services throughthe public, worldwide packet-switching communication network of networksnow commonly referred to as the Internet 1032.

A computer called a server host 1034 connected to the Internet 1032hosts a process that provides a service in response to informationreceived over the Internet 1032. For example, server host 1034 hosts aprocess that provides information representing video data forpresentation at display 1010. It is contemplated that the components ofsystem 1000 can be deployed in various configurations within othercomputer systems, e.g., host and server.

At least some embodiments of the present disclosure are related to theuse of computer system 1000 for implementing some or all of thetechniques described herein. According to one embodiment, thosetechniques are performed by computer system 1000 in response toprocessing unit 1012 executing one or more sequences of one or moreprocessor instructions contained in memory 1004. Such instructions, alsocalled computer instructions, software and program code, may be readinto memory 1004 from another computer-readable medium 1006 such asstorage device or network link. Execution of the sequences ofinstructions contained in memory 1004 causes processing unit 1012 toperform one or more of the method steps described herein. In alternativeembodiments, hardware, such as ASIC, may be used in place of or incombination with software. Thus, embodiments of the present disclosureare not limited to any specific combination of hardware and software,unless otherwise explicitly stated herein.

The signals transmitted over network link and other networks throughcommunications interface, carry information to and from computer system1000. Computer system 1000 can send and receive information, includingprogram code, through the networks, among others, through network linkand communications interface. In an example using the Internet, a serverhost transmits program code for a particular application, requested by amessage sent from computer, through Internet, ISP equipment, localnetwork and communications interface. The received code may be executedby processor 1002 as it is received, or may be stored in memory 1004 orin storage device or other non-volatile storage for later execution, orboth.

For the purposes of this disclosure a module is a software, hardware, orfirmware (or combinations thereof) system, process or functionality, orcomponent thereof, that performs or facilitates the processes, features,and/or functions described herein (with or without human interaction oraugmentation). A module can include sub-modules. Software components ofa module may be stored on a computer readable medium for execution by aprocessor. Modules may be integral to one or more servers, or be loadedand executed by one or more servers. One or more modules may be groupedinto an engine or an application.

For the purposes of this disclosure the term “user”, “subscriber”“consumer” or “customer” should be understood to refer to a consumer ofdata supplied by a data provider. By way of example, and not limitation,the term “user” or “subscriber” can refer to a person who receives dataprovided by the data or service provider over the Internet in a browsersession, or can refer to an automated software application whichreceives the data and stores or processes the data.

Those skilled in the art will recognize that the methods and systems ofthe present disclosure may be implemented in many manners and as suchare not to be limited by the foregoing exemplary embodiments andexamples. In other words, functional elements being performed by singleor multiple components, in various combinations of hardware and softwareor firmware, and individual functions, may be distributed among softwareapplications at either the client level or server level or both. In thisregard, any number of the features of the different embodimentsdescribed herein may be combined into single or multiple embodiments,and alternate embodiments having fewer than, or more than, all of thefeatures described herein are possible.

Functionality may also be, in whole or in part, distributed amongmultiple components, in manners now known or to become known. Thus,myriad software/hardware/firmware combinations are possible in achievingthe functions, features, interfaces and preferences described herein.Moreover, the scope of the present disclosure covers conventionallyknown manners for carrying out the described features and functions andinterfaces, as well as those variations and modifications that may bemade to the hardware or software or firmware components described hereinas would be understood by those skilled in the art now and hereafter.

Furthermore, the embodiments of methods presented and described asflowcharts in this disclosure are provided by way of example in order toprovide a more complete understanding of the technology. The disclosedmethods are not limited to the operations and logical flow presentedherein. Alternative embodiments are contemplated in which the order ofthe various operations is altered and in which sub-operations describedas being part of a larger operation are performed independently.

While various embodiments have been described for purposes of thisdisclosure, such embodiments should not be deemed to limit the teachingof this disclosure to those embodiments. Various changes andmodifications may be made to the elements and operations described aboveto obtain a result that remains within the scope of the systems andprocesses described in this disclosure.

What is claimed is:
 1. A method comprising: identifying, via a computing device, devices concurrently present at a location in a structure; determining, via the computing device, a Bluetooth Low Energy (BLE) signal strength associated with each identified device, said BLE signal strength determined from received BLE signals from each identified device; determining, via the computing device, an audio-distance measurement associated with each identified device, said audio-distance measurement based on acoustic signals and time stamps communicated to and from each identified device; receiving, at the computing device, an inertia measurement from each device; and determining, via the computing device, a three-dimensional (3D) spatial mapping of said location based on, for each identified device, the BLE signal strength, the audio-distance measurement and the inertia measurement, said 3D spatial mapping comprising identifiers and positional information of each identified device within said location.
 2. The method of claim 1, wherein said determination of the 3D spatial mapping comprises fusing said BLE signal strength, audio-distance measurement and the inertia measurement of each device.
 3. The method of claim 1, further comprising: broadcasting, via the computing device, said determined 3D spatial mapping of the location to each identified device at said location.
 4. The method of claim 1, wherein said inertia measurement for each device is based on a constant rotational reference associated with said location, said rotational reference is associated with a magnetometer associated with said location.
 5. The method of claim 1, wherein said inertia measurement for each device comprises at least one of orientation and motion information.
 6. The method of claim 1, wherein said determining said audio-distance measurement based on said acoustic signals comprises: receiving, by at least one microphone of the computing device, a first acoustic signal from a microphone of each identified device; and communicating, from at least one speaker of the computing device, a second acoustic signal for reception by a microphone of each identified device.
 7. The method of claim 6, further comprising: receiving, at the computing device, a time stamp associated with the second acoustic signal from each identified device, said second acoustic signal timestamp associated with a time the second acoustic signal was received by each identified device; and communicating, via the computing device, a time stamp associated with the first acoustic signal to each identified device, said first acoustic signal time stamp associated with a time the first acoustic signal was received by the computing device.
 8. The method of claim 1, wherein said BLE signal strength determination is based on said determined audio-distance measurement, wherein said BLE signal strength for an identified device is refined in accordance with an audio-distance measurement associated with said identified device.
 9. The method of claim 1, wherein said collocation of devices at the location comprises each identified device being within a predetermined distance to the computing device within said location.
 10. The method of claim 1, wherein said identification of devices at said location comprises identifying which devices are emitting a BLE signal.
 11. The method of claim 1, further comprising: receiving, at the computing device, a selection of a device identified in said 3D spatial mapping, said selection enabling communication with the selected device.
 12. The method of claim 11, wherein said communication occurs over at least one of a network associated with said location and via a BLE signal packet communicated with the selected device.
 13. The method of claim 11, further comprising: determining, via the computing device, the 3D spatial mapping of said location based on, for each identified device, the audio-distance measurement.
 14. The method of claim 1, wherein said 3D spatial mapping is updated upon a change in said positional information of at least one device determined to be at said location.
 15. A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions, that when executed by a processor associated with a computing device, performs a method comprising: identifying devices concurrently present at a location in a structure; determining a Bluetooth Low Energy (BLE) signal strength associated with each identified device, said BLE signal strength determined from received BLE signals from each identified device; determining an audio-distance measurement associated with each identified device, said audio-distance measurement based on acoustic signals and time stamps communicated to and from each identified device; receiving an inertia measurement from each device; determining a three-dimensional (3D) spatial mapping of said location based on, for each identified device, the BLE signal strength, the audio-distance measurement and the inertia measurement, said 3D spatial mapping comprising identifiers and positional information of each identified device within said location.
 16. The non-transitory computer-readable storage medium of claim 15, wherein said determination of the 3D spatial mapping comprises fusing said BLE signal strength, audio-distance measurement and the inertia measurement of each device.
 17. The non-transitory computer-readable storage medium of claim 15, further comprising: broadcasting said determined 3D spatial mapping of the location to each identified device at said location; and receiving a selection of a device identified in said 3D spatial mapping, said selection enabling communication with the selected device.
 18. The non-transitory computer-readable storage medium of claim 15, further comprising: wherein said determining said audio-distance measurement based on said acoustic signals comprises: receiving a first acoustic signal from at least one microphone of each identified device; and communicating a second acoustic signal for reception by at least one microphone of each identified device; receiving a time stamp associated with the second acoustic signal from each identified device, said second acoustic signal timestamp associated with a time the second acoustic signal was received by each identified device; and communicating a time stamp associated with the first acoustic signal to each identified device, said first acoustic signal time stamp associated with a time the first acoustic signal was received by the computing device.
 19. The non-transitory computer-readable storage medium of claim 15, wherein said BLE signal strength determination is based on said determined audio-distance measurement, wherein said BLE signal strength for an identified device is refined in accordance with an audio-distance measurement associated with said identified device.
 20. A system comprising: a processor; a non-transitory computer-readable storage medium for tangibly storing thereon program logic for execution by the processor, the program logic comprising: identification logic executed by the processor for identifying devices concurrently present at a location in a structure; determination logic executed by the processor for determining a Bluetooth Low Energy (BLE) signal strength associated with each identified device, said BLE signal strength determined from received BLE signals from each identified device; determination logic executed by the processor for determining an audio-distance measurement associated with each identified device, said audio-distance measurement based on acoustic signals and time stamps communicated to and from each identified device; receiving logic executed by the processor for receiving an inertia measurement from each device; determination logic executed by the processor for determining a three-dimensional (3D) spatial mapping of said location based on, for each identified device, the BLE signal strength, the audio-distance measurement and the inertia measurement, said 3D spatial mapping comprising identifiers and positional information of each identified device within said location; and communication logic executed by the processor for broadcasting said determined 3D spatial mapping of the location to each identified device at said location. 